Gang Scheduling Extensions for I/O Intensive Workloads

Scientific applications are becoming more I/O demanding than ever. For such applications, the system with dedicated I/O nodes does not provide enough scalability. Rather, a serverless approach is a viable alternative. However, with the serverless approach, a job’s execution time is decided by whether it is co-located with the file blocks it needs. Gang scheduling (GS), which is used in supercomputing centers to schedule parallel jobs, is completely not aware of the application’s spatial preferences. In this paper, we show that gang scheduling delivers poor performance towards workloads with high I/O intensities (I/O ratio higher than 50%). We propose an I/O-aware extension of gang scheduling, IOGS, which co-locates jobs with their files. While IOGS performs better for high I/O intensity workloads, its performance for workloads with lower I/O intensities is rather poor because of high system fragmentation. Further, we propose an adaptive strategy, adaptive-IOGS, which attempts to combine the advantages of both gang scheduling and GS, and we show that adaptive-IOGS is better than the other two schemes in many scenarios. Finally, we combine process migration techniques with adaptive-IOGS, and propose Migration-IOGS, which is shown to be the best among the four for a wide spectrum of workloads.